Navigable telepresence method and system utilizing an array of cameras

ABSTRACT

A telepresence system for providing users with displays of a remote environment. In certain embodiments, the system includes a plurality of arrays of cameras, with each arrays are situated at varying lengths from the environment. Users navigate the arrays via interface devices. The system interprets the user inputs and selects camera outputs based thereon. Multiple users are able to navigate simultaneously and independently through the environment. In embodiments having overlapping camera views, the system effectuates seamless motion along camera paths by processing the camera outputs corresponding to the views.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is a continuation of commonly assigned pendingU.S. patent application Ser. No. 09/283,413 filed on Apr. 1, 1999,entitled A Navigable Telepresence Method And System Utilizing An Arrayof Cameras and claims priority under 35 U.S.C. §120 to such applicationwhich claims the benefit of U.S. Provisional Application Serial No.60/080,413, filed on Apr. 2, 1998, both of which are herein incorporatedby reference.

BACKGROUND OF THE INVENTION

[0002] 1. Field Of The Invention

[0003] The present invention relates to a telepresence system and, moreparticularly, to a navigable camera array telepresence system and methodof using same.

[0004] 2. Description Of Related Art

[0005] In general, a need exists for the development of telepresencesystems suitable for use with static venues, such as museums, anddynamic venues or events, such as a music concerts. The viewing of suchvenues is limited by time, geographical location, and the viewercapacity of the venue. For example, potential visitors to a museum maybe prevented from viewing an exhibit due to the limited hours the museumis open. Similarly, music concert producers must turn back fans due tothe limited seating of an arena. In short, limited access to venuesreduces the revenue generated.

[0006] In an attempt to increase the revenue stream from both static anddynamic venues, such venues have been recorded for broadcast ordistribution. In some instances, dynamic venues are also broadcast live.While such broadcasting increases access to the venues, it involvesconsiderable production effort. Typically, recorded broadcasts must becut and edited, as views from multiple cameras are pieced together.These editorial and production efforts are costly.

[0007] In some instances, the broadcast resulting from these editorialand production efforts provides viewers with limited enjoyment.Specifically, the broadcast is typically based on filming the venue froma finite number of predetermined cameras. Thus, the broadcast containslimited viewing angles and perspectives of the venue. Moreover, theviewing angles and perspectives presented in the broadcast are thoseselected by a producer or director during the editorial and productionprocess; there is no viewer autonomy. Furthermore, although thebroadcast is often recorded for multiple viewings, the broadcast haslimited content life because each viewing is identical to the first.Because each showing looks and sounds the same, viewers rarely come backfor multiple viewings.

[0008] A viewer fortunate enough to attend a venue in person willencounter many of the same problems. For example, a museum-goer mustremain behind the barricades, viewing exhibits from limited angles andperspectives. Similarly, concert-goers are often restricted to aparticular seat or section in an arena. Even if a viewer were allowedfree access to the entire arena to videotape the venue, such a recordingwould also have limited content life because each viewing would be thesame as the first. Therefore, a need exists for a telepresence systemthat preferably provides user autonomy while resulting in recordingswith enhanced content life at a reduced production cost.

[0009] Apparently, attempts have been made to develop telepresencesystems to satisfy some of the foregoing needs. One telepresence systemis described in U.S. Pat. No. 5,708,469 for Multiple View TelepresenceCamera Systems Using A Wire Cage Which Surrounds A Polarity Of MultipleCameras And Identifies The Fields Of View, issued Jan. 13, 1998. Thesystem disclosed therein includes a plurality of cameras, wherein eachcamera has a field of view that is space-contiguous with and at a rightangle to at least one other camera. In other words, it is preferablethat the camera fields of view do not overlap each other. A userinterface allows the user to jump between views. In order for the user'sview to move through the venue or environment, a moving vehicle carriesthe cameras.

[0010] This system, however, has several drawbacks. For example, inorder for a viewer's perspective to move through the venue, the movingvehicle must be actuated and controlled. In this regard, operation ofthe system is complicated. Furthermore, because the camera views arecontiguous, typically at right angles, changing camera views results ina discontinuous image.

[0011] Other attempts at providing a telepresence system have taken theform of a 360 degree camera systems. One such system is described inU.S. Pat. No. 5,745,305 for Panoramic Viewing Apparatus, issued Apr. 281998. The system described therein provides a 360 degree view ofenvironment by arranging multiple cameras around a pyramid shapedreflective element. Each camera, all of which share a common virtualoptical center, receives an image from a different side of thereflective pyramid. Other types of 360 degree camera systems employ aparabolic lens or a rotating camera.

[0012] Such 360 degree camera systems also suffer from drawbacks. Inparticular, such systems limit the user's view to 360 degrees from agiven point perspective. In other words, 360 degree camera systemsprovide the user with a panoramic view from a single location. Only ifthe camera system was mounted on a moving vehicle could the userexperience simulated movement through an environment.

[0013] U.S. Pat. No. 5,187,571 for Television System For DisplayingMultiple Views of A Remote Location issued Feb. 16, 1993, describes acamera system similar to the 360 degree camera systems described above.The system described provides a user to select an arbitrary andcontinuously variable section of an aggregate field of view. Multiplecameras are aligned so that each camera's field of view mergescontiguously with those of adjacent cameras thereby creating theaggregate field of view. The aggregate field of view may expand to cover360 degrees. In order to create the aggregate field of view, thecameras' views must be contiguous. In order for the camera views to becontiguous, the cameras have to share a common point perspective, orvertex. Thus, like the previously described 360 degree camera systems,the system of U.S. Pat. No. 5,187,571 limits a user's view to a singlepoint perspective, rather than allowing a user to experience movement inperspective through an environment.

[0014] Also, with regard to the system of U.S. Pat. No. 5,187,571, inorder to achieve the contiguity between camera views, a relativelycomplex arrangement of mirrors is required. Additionally, each cameraseemingly must also be placed in the same vertical plane.

[0015] Thus, a need still exists for an improved telepresence systemthat provides the ability to better simulate a viewer's actual presencein a venue, preferably in real time.

SUMMARY OF THE INVENTION

[0016] These and other needs are satisfied by the present invention. Atelepresence system according to one embodiment of the present inventionincludes an array of cameras, each of which has an associated view of anenvironment and an associated output representing the view. The systemalso includes a first user interface device having first user inputsassociated with movement along a first path in the array. The systemfurther includes a second user interface device having second userinputs associated with movement along a second path in the array. Aprocessing element is coupled to the user interface devices. Theprocessing element receives and interprets the first inputs and selectsoutputs of cameras in the first path. Similarly, the processing elementreceives and interprets the second inputs and selects outputs of camerasin the second path independently of the first inputs. Thus, a first userand a second user are able to navigate simultaneously and independentlythrough the array. In another embodiment, the system may also mix theoutput by mosaicing or tweening the output images. In a furtherembodiment of the present invention the telepresence systemdistinguishes between permissible cameras in the array and impermissiblecameras in the array. In yet another embodiment of the present inventionthe telepresence system allows a user to move forward or backwardthrough the environment, which provides the user the opportunity to moveforward or backward through the environment.

BRIEF DESCRIPTION OF THE DRAWINGS

[0017]FIG. 1 is an overall schematic of one embodiment of the presentinvention.

[0018]FIG. 2a is a perspective view of a camera and a camera railsection of the array according to one embodiment of the presentinvention.

[0019]FIGS. 2b-2 d are side plan views of a camera and a camera railaccording to one embodiment of the present invention.

[0020]FIG. 2e is a top plan view of a camera rail according to oneembodiment of the present invention.

[0021]FIG. 3 is a perspective view of a portion of the camera arrayaccording to one embodiment of the present invention.

[0022]FIG. 4 is a perspective view of a portion of the camera arrayaccording to an alternate embodiment of the present invention.

[0023]FIG. 5 is a flowchart illustrating the general operation of theuser interface according to one embodiment of the present invention.

[0024]FIG. 6 is a flowchart illustrating in detail a portion of theoperation shown in FIG. 5.

[0025]FIG. 7a is a perspective view of a portion of one embodiment ofthe present invention illustrating the arrangement of the camera arrayrelative to objects being viewed.

[0026]FIGS. 7b-7 g illustrate views from the perspectives of selectedcameras of the array in FIG. 7a.

[0027]FIG. 8 is a schematic view of an alternate embodiment of thepresent invention.

[0028]FIG. 9 is a schematic view of a server according to one embodimentof the present invention.

[0029]FIG. 10 is a schematic view of a server according to an alternateembodiment of the present invention.

[0030]FIG. 11 is a top plan view of an alternate embodiment of thepresent invention.

[0031]FIG. 12 is a flowchart illustrating in detail the image captureportion of the operation of the embodiment shown in FIG. 11.

DESCRIPTION OF PREFERRED EMBODIMENTS

[0032] 1. General Description Of Preferred Embodiments

[0033] The present invention relates to a telepresence system that, in apreferred embodiments, uses modular, interlocking arrays ofmicrocameras. The cameras are on rails, with each rail holding aplurality of cameras. These cameras, each locked in a fixed relation toevery adjacent camera on the array and dispersed dimensionally in agiven environment, transmit image output to an associated storage node,thereby enabling remote viewers to navigate through such environmentwith the same moving light reflections and shadows) that characterize anactual in-environment transit.

[0034] In another preferred embodiment, the outputs of thesemicrocameras are linked by tiny (less than half the width of a humanhair) Vertical Cavity Surface Emitting Lasers (VCSELs) to opticalfibers, fed through area net hubs, buffered on server arrays or serverfarms (either for recording or (instantaneous) relay) and sent toviewers at remote terminals, interactive wall screens, or mobile imageappliances (like Virtual Retinal Displays). Each remote viewer, throughan intuitive graphical user interface (GUI), can navigate effortlesslythrough the environment, enabling seamless movement through the event.

[0035] This involves a multiplexed, electronic switching process(invisible to the viewer) which moves the viewer's point perspectivefrom camera to camera. Rather than relying, per se, on physically movinga microcamera through space, the system uses the multiplicity ofpositioned microcameras to move the viewer's perspective frommicrocamera node to adjacent microcamera node in a way that provides theviewer with a sequential visual and acoustical path throughout theextent of the array. This allows the viewer to fluidly track or dollythrough a 3-dimensional remote environment, to move through an event andmake autonomous real-time decisions about where to move and when tolinger.

[0036] Instead of investing the viewer with the capacity to physicallymove a robotic camera, which would immediately limit the number ofviewers that could simultaneously control their own course and navigatevia storage nodes containing images of an environment associated with apre-existing array of cameras. The user can move around the environmentin any direction—clockwise or counterclockwise, up, down, closer to orfurther away from the environment, or some combination thereof.Moreover, image output mixing, such as mosaicing and tweening,effectuates seamless motion throughout the environment.

[0037] 2. Detailed Description Of Preferred Embodiments

[0038] Certain embodiments of the present invention will now bedescribed in greater detail with reference to the drawings. It isunderstood that the operation and functionality of many of thecomponents of the embodiments described herein are known to one skilledin the art and, as such, the present description does not go into detailinto such operative and functionality.

[0039] A telepresence system 100 according to the present invention isshown in FIG. 1. The telepresence system 100 generally includes an array10 of cameras 14 coupled to a server 18, which in turn is coupled to oneor more users 22 each having a user interfaced/display device 24. Aswill be understood to one skilled it the art, the operation andfunctionality of the embodiment described herein is provided, in part,by the server and user interface/display device. While the operation ofthese components is not described by way of particular code listings orlogic diagrams, it is to be understood that one skilled in the art willbe able to arrive at suitable implementations based on the functionaland operational details provided herein. Furthermore, the scope of thepresent invention is not to be construed as limited to any particularcode or logic implementation.

[0040] In the present embodiment, the camera array 10 is conceptualizedas being in an X, Z coordinate system. This allows each camera to havean associated, unique node address comprising an X, and Z coordinate (X,Z). In the present embodiment, for example, a coordinate valuecorresponding to an axis of a particular camera represents the number ofcamera positions along that axis the particular camera is displaced froma reference camera. In the present embodiment, from the user'sperspective the X axis runs left and right, and the Z axis runs down andup. Each camera 14 is identified by its X, Z coordinate. It is to beunderstood, however, that other methods of identifying cameras 14 can beused. For example, other coordinate systems, such as those notingangular displacement from a fixed reference point as well as coordinatesystems that indicate relative displacement from the current camera nodemay be used. In another alternate embodiment, the array is threedimensional, located in an X, Y, Z coordinate system.

[0041] The array 10 comprises a plurality of rails 12, each rail 12including a series of cameras 14. In the present preferred embodiment,the cameras 14 are microcameras. The output from the microcameras 14 arecoupled to the server 18 by means of local area hubs 16. The local areahubs 16 gather the outputs and, when necessary, amplify the outputs fortransmission to the server 18. In an alternate embodiment, the localarea hubs 16 multiplex the outputs for transmission to the server 18.Although the figure depicts the communication links 15 between thecameras 14 and the server 18 as being hardwired, it is to be understoodthat wireless links may be employed. Thus, it is within the scope of thepresent invention for the communication links 15 to take the form offiber optics, cable, satellite, microwave transmission, internet, andthe like.

[0042] Also coupled to the server 18 is an electronic storage device 20.The server 18 transfers the outputs to the electronic storage device 20.The electronic (mass) storage device 20, in turn, transfers eachcamera's output onto a storage medium or means, such as CD-ROM, DVD,tape, platter, disk array, or the like. The output of each camera 14 isstored in a particular location on the storage medium associated withthat camera 14 or is stored with an indication to which camera 14 eachstored output corresponds. For example, the output of each camera 14 isstored in contiguous locations on a separate disk, tape, CD-ROM, orplatter. As is known in the art, the camera output may be stored in acompressed format, such as JPEG, MPEG1, MPEG2, and the like. Havingstored each output allows a user to later view the environment over andover again, each time moving through the array 10 in a new path, asdescribed below. In some embodiments of the present invention, such asthose providing only real-time viewing, no storage device is required.

[0043] As will be described in detail below, the server 18 receivesoutput from the cameras 14 in the array. The server 18 processes theseoutputs for either storage in the electronic storage device 20,transmission to the users 22 or both.

[0044] It is to be understood that although the server 18 is configuredto provide the functionality of the system 100 in the presentembodiment, it is to be understood that other processing elements mayprovide the functionality of the system 100. For example, in alternateembodiments, the user interface device is a personal computer programmedto interpret the user input and transmit an indication of the desiredcurrent node address, buffer outputs from the array, and provide otherof the described functions.

[0045] As shown, the system 100 can accommodate (but does not require)multiple users 22. Each user 22 has associated therewith a userinterface device including a user display device (collectively 24). Forexample, user 22-1 has an associated user interface device and a userdisplay device in the form of a computer 24-1 having a monitor and akeyboard. User 22-2 has associated therewith an interactive wall screen24-2 which serves as a user interface device and a user display device.The user interface device and the user display device of user 22-3includes a mobile audio and image appliance 24-3. A digital interactiveTV 24-4 is the user interface device and user display device of user22-4. Similarly, user 22-5 has a voice recognition unit and monitor 24-5as the user interface and display devices. It is to be understood thatthe foregoing user interface devices and user display devices are merelyexemplary; for example, other interface devices include a mouse, touchscreen, biofeedback devices, as well as those identified in U.S.Provisional Patent Application Serial No. 60/080,413 and the like.

[0046] As described in detail below, each user interface device 24 hasassociated therewith user inputs. These user inputs allow each user 22to move or navigate independently through the array 10. In other words,each user 22 enters inputs to generally select which camera outputs aretransferred to the user display device. Preferably, each user displaydevice includes a graphical representation of the array 10. Thegraphical representation includes an indication of which camera in thearray the output of which is being viewed. The user inputs allow eachuser to not only select particular cameras, but also to select relativemovement or navigational paths through the array 10.

[0047] As shown in FIG. 1, each user 22 may be coupled to the server 18by an independent communication link. Furthermore, each communicationlink may employ different technology. For example, in alternateembodiments, the communication links include an internet link, amicrowave signal link, a satellite link, a cable link, a fiber opticlink, a wireless link, and the like.

[0048] It is to be understood that the array 10 provides severaladvantages. For example, because the array 10 employs a series ofcameras 14, no individual camera, or the entire array 10 for thatmatter, need be moved in order to obtain a seamless view of theenvironment. Instead, the user navigates through the array 10, which isstrategically placed through and around the physical environment to beviewed. Furthermore, because the cameras 14 of the array 10 arephysically located at different points in the environment to be viewed,a user is able to view changes in perspective, a feature unavailable toa single camera that merely changes focal length.

[0049] Microcameras

[0050] Each camera 14 is preferably a microcamera. Themicrocameras—microlenses mounted on thumbnail-sized CMOS active pixelsensor (APS) microchips—are arranged in patterns that enable viewers tomove radically, in straight lines, or in fluid combinations thereof. Thecameras are produced in a mainstream manufacturing process, by severalcompanies, including Photobit, Pasadena, Calif.; Sarnoff Corporation,Princeton, N.J.; and VLSI Vision, Ltd., Edinburgh, Scotland.

[0051] Structure of the Array

[0052] The structure of the array 10 will now be described in greaterdetail with reference to FIGS. 2a-2 e. In general, the camera array 10of the present embodiment comprises a series of modular rails 12carrying microcameras 14. The structure of the rails 12 and cameras 14will now be discussed in greater detail with reference to FIGS. 2athrough 2 d. Each camera 14 includes registration pins 34. In thepreferred embodiment, the cameras 14 utilize VCSELs to transfer theiroutputs to the rail 12. It is to be understood that the presentinvention is not limited to any particular type of camera 14, however,or even to an array 10 consisting of only one type of camera 14.

[0053] Each rail 12 includes two sides, 12 a, 12 b, at least one ofwhich 12 b is hingeably connected to the base 12 c of the rail 12. Thebase 12 c includes docking ports 36 for receiving the registration pins34 of the camera 14. When the camera 14 is seated on a rail 12 such thatthe registration pins 34 are fully engaged in the docking ports 36, thehinged side 12 b of the rail 12 is moved against the base 32 of thecamera 14, thereby securing the camera 14 to the rail 12.

[0054] Each rail 12 further includes a first end 38 and a second end 44.The first end 38 includes, in the present embodiment, two locking pins40 and a protected transmission relay port 42 for transmitting thecamera outputs. The second end 44 includes two guide holes 46 forreceiving the locking pins 40, and a transmission receiving port 48.Thus, the first end 38 of one rail 12 is engagable with a second end 44of another rail 12. Therefore, each rail 12 is modular and can befunctionally connected to another rail to create the array 10.

[0055] Once the camera 14 is securely seated to the rail 12, the camera14 is positioned such that the camera output may be transmitted via theVCSEL to the rail 12. Each rail 12 includes communication paths fortransmitting the output from each camera 14.

[0056] Although the array 10 is shown having a particular configuration,it is to be understood that virtually any configuration of rails 12 andcameras 14 is within the scope of the present invention. For example,the array 10 may be a linear array of cameras 14, a 2-dimensional arrayof cameras 14, a 3-dimensional array of cameras 14, or any combinationthereof. Furthermore, the array 10 need not be comprised solely oflinear segments, but rather may include curvilinear sections.

[0057] The array 10 is supported by any of a number of support means.For example, the array 10 can be fixedly mounted to a wall or ceiling;the array 10 can be secured to a moveable frame that can be wheeled intoposition in the environment or supported from cables.

[0058]FIG. 3 illustrates an example of a portion of the array 10. Asshown, the array 10 comprises five rows of rails 12 a, through 12 e.Each of these rails 12 a-12 e is directed towards a central plane, whichsubstantially passes through the center row 12 c. Consequently, for anyobject placed in the same plane as the middle row 12 c, a user would beable to view the object essentially from the bottom, front, and top.

[0059] As noted above, the rails 12 of the array 10 need not have thesame geometry. For example, some of the rails 12 may be straight whileothers may be curved. For example, FIG. 4 illustrates the cameraalignment that results from utilizing curved rails. It should be notedthat rails in FIG. 4 have been made transparent so that the arrangementof cameras 14 may be easily seen.

[0060] In an alternate embodiment, each rail is configured in astep-like fashion or an arc with each camera above (or below) and infront of a previous camera. In such an arrangement, the user has theoption of moving forward through the environment.

[0061] It is to be understood that the spacing of the microcameras 14depends on the particular application, including the objects beingviewed, the focal length of the microcameras 14, and the speed ofmovement through the array 10. In one embodiment the distance betweenmicrocameras 14 can be approximated by analogy to a conventional moviereel recording projector. In general, the speed of movement of aprojector through an environment divided by the frames per unit of timesecond results in a frame-distance ratio.

[0062] For example, as shown by the following equations, in someapplications a frame is taken ever inch. A conventional movie projectorrecords twenty-four frames per second. When such a projector is movedthrough an environment at two feet per second, a frame is takenapproximately every inch.${\frac{2\quad {ft}}{\sec} \div \frac{24\quad {frames}}{\sec}} = {\frac{2\quad {ft}}{24\quad {frames}} = {\frac{1\quad {ft}}{12\quad {frames}} = {\frac{12\quad {inches}}{12\quad {frames}} = {\frac{1\quad {inch}}{1\quad {frame}} = {1\quad {frame}\quad {per}\quad {{inch}.}}}}}}$

[0063] A frame of the projector is analogous to a camera 14 in thepresent invention. Thus, where one frame per inch results in a moviehaving a seamless view of the environment, so too does one camera 14 perinch. Thus, in one embodiment of the present invention the cameras 14are spaced approximately one inch apart, thereby resulting in a seamlessview of the environment.

[0064] Navigation Through the System

[0065] The general operation of the present embodiment will now bedescribed with reference to FIG. 5 and continuing reference to FIG. 1.As shown in step 110, the user is presented with a predeterminedstarting view of the environment corresponding to a starting camera. Itis to be understood that the operation of the system is controlled, inpart, by software residing in the server. As noted above, the systemassociates each camera in the array with a coordinate. Thus, the systemis able to note the coordinates of the starting camera node. The cameraoutput and, thus the corresponding view, changes only upon receiving auser input.

[0066] When the user determines that they want to move or navigatethrough the array, the user enters a user input through the userinterface device 24. As described below, the user inputs of the presentembodiment generally include moving to the right, to the left, up, ordown in the array. Additionally, a user may jump to a particular camerain the array. In alternate embodiments, a subset of these or otherinputs, such as forward, backward, diagonal, over, and under, are used.The user interface device, in turn, transmits the user input to theserver in step 120.

[0067] Next, the server receives the user input in step 130 and proceedsto decode the input. In the present embodiment, decoding the inputgenerally involves determining whether the user wishes to move to theright, to the left, up, or down in the array.

[0068] On the other hand, if the received user input does not correspondto backward, then The server 18 proceeds to determine whether the inputcorresponds to moving to the user's right in the array 10. Thisdetermination is shown in step 140. If the received user input doescorrespond to moving to the right, the current node address isincremented along the X axis in step 150 to obtain an updated address.

[0069] If the received user input does not correspond to moving to theright in the array, the server 18 then determines whether the inputcorresponds to moving to the user's left in the array 10 in step 160.Upon determining that the input does correspond to moving to the left,the server 18 then decrements the current node address along the X axisto arrive at the updated address. This is shown in step 170.

[0070] If the received user input does not correspond to either movingto the right or to the left, the server 18 then determines whether theinput corresponds to moving up in the array. This determination is madein step 180. If the user input corresponds to moving up, in step 190,the server 18 increments the current node address along the Z axis,thereby obtaining an updated address.

[0071] Next, the server 18 determines whether the received user inputcorresponds to moving down in the array 10. This determination is madein step 200. If the input does correspond to moving down in the array10, in step 210 the server 18 decrements the current node address alongthe Z axis.

[0072] Lastly, in step 220 the server 18 determines whether the receiveduser input corresponds to jumping or changing the view to a particularcamera 14. As indicated in FIG. 5, if the input corresponds to jumpingto a particular camera 14, the server 18 changes the current nodeaddress to reflect the desired camera position. Updating the nodeaddress is shown as step 230. In an alternate embodiment, the inputcorresponds to jumping to a particular position in the array 10, notidentified by the user as being a particular camera but by somereference to the venue, such as stage right.

[0073] It is to be understood that the server 18 may decode the receiveduser inputs in any of a number of ways, including in any order. Forexample, in an alternate embodiment the server 18 first determineswhether the user input corresponds to up or down. In another alternate,preferred embodiment, user navigation includes moving forward, backward,to the left and right, and up and down through a three dimensionalarray.

[0074] If the received user input does not correspond to any of therecognized inputs, namely to the right, to the left, up, down, orjumping to a particular position in the array 10 then in step 240, theserver 18 causes a message signal to be transmitted to the user displaydevice 24, causing a message to be displayed to the user 22 that thereceived input was not understood. Operation of the system 100 thencontinues with step 120, and the server 18 awaits receipt of the nextuser input.

[0075] After adjusting the current node address, either by incrementingor decrementing the node address along an axis or by jumping to aparticular node address, the server 18 proceeds in step 250 to adjustthe user's view. Once the view is adjusted, operation of the system 100continues again with step 120 as the server 18 awaits receipt of thenext user input.

[0076] In an alternate embodiment, the server 18 continues to update thenode address and adjust the view based on the received user input. Forexample, if the user input corresponded to “moving to the right”, thenoperation of the system 100 would continuously loop through steps 140,150, and 250, checking for a different input. When the different inputis received, the server 18 continuously updates the view accordingly.

[0077] It is to be understood that the foregoing user inputs, namely, tothe right, to the left, up, and down, are merely general descriptions ofmovement through the array. Although the present invention is not solimited, in the present preferred embodiment, movement in each of thesegeneral directions is further defined based upon the user input.

[0078] Accordingly, FIG. 6 is a more detailed diagram of the operationof the system according to steps 140, 150, and 250 of FIG. 5. Moreover,it is to be understood that while FIG. 6 describes more detailedmovement one direction i.e., to the right, the same detailed movementcan be applied in any other direction. As illustrated, the determinationof whether the user input corresponds to moving to the right actuallyinvolves several determinations. As described in detail below, thesedeterminations include moving to the right through the array 10 atdifferent speeds, moving to the right into a composited additionalsource output at different speeds, and having the user input overriddenby the system 100.

[0079] The present invention allows a user 22 to navigate through thearray 10 at the different speeds. Depending on the speed (i.e. number ofcamera nodes transversed per unit of time) indicated by the user'sinput, such as movement of a pointing device (or other interfacedevice), the server 18 will apply an algorithm that controls thetransition between camera outputs either at critical speed (n nodes/perunit of time), under critical speed (n−1 nodes/per unit of time), orover critical speed (n+1 nodes/per unit of time).

[0080] It is to be understood that speed of movement through the array10 can alternatively be expressed as the time to switch from one camera14 to another camera 14.

[0081] Specifically, as shown in step 140 a, the server 18 makes thedetermination whether the user input corresponds to moving to the rightat a critical speed. The critical speed is preferably a predeterminedspeed of movement through the array 10 set by the system operator ordesigner depending on the anticipated environment being viewed. Further,the critical speed depends upon various other factors, such as focallength, distance between cameras, distance between the cameras and theviewed object, and the like. The speed of movement through the array 10is controlled by the number of cameras 14 traversed in a given timeperiod. Thus, the movement through the array 10 at critical speedcorresponds to traversing some number, “n”, camera nodes permillisecond, or taking some amount of time, “s”, to switch from onecamera 14 to another. It is to be understood that in the same embodimentthe critical speed of moving through the array 10 in one dimension neednot equal the critical speed of moving through the array in anotherdimension. Consequently, the server 18 increments the current nodeaddress along the X axis at n nodes per millisecond.

[0082] In the present preferred embodiment the user traversestwenty-four cameras 14 per second. As discussed above, a movie projectorrecords twenty-four frames per second. Analogizing between the movieprojector and the present invention, at critical the user traverses (andthe server 18 switches between) approximately twenty-four cameras 14 persecond, or a camera 14 approximately every 0.04167 seconds.

[0083] As shown in FIG. 6, the user 22 may advance not only at criticalspeed, but also at over the critical speed, as shown in step 140 b, orat under the critical speed, as shown in step 140 c. Where the userinput “I” indicates movement through the array 10 at over the criticalspeed, the server 18 increments the current node address along the Xaxis by a unit of greater than n, for example, at n+2 nodes permillisecond. The step of incrementing the current node address at n+1nodes per millisecond along the X axis is shown in step 150 b. Where theuser input “I” indicates movement through the array 10 at under thecritical speed, the server 18 proceeds to increment the current nodeaddress at a variable less than n, for example, at n−1 nodes permillisecond. This operation is shown as step 150 c.

[0084] Scaleable Arrays

[0085] The shape of the array 10 can also be electronically scaled andthe system 100 designed with a “center of gravity” that will ease auser's image path back to a “starting” or “critical position” node orring of nodes, either when the user 22 releases control or when thesystem 100 is programmed to override the user's autonomy; that is tosay, the active perimeter or geometry of the array 10 can bepre-configured to change at specified times or intervals in order tocorral or focus attention in a situation that requires dramatic shaping.The system operator can, by real-time manipulation or via apre-configured electronic proxy sequentially activate or deactivatedesignated portions of the camera array 10. This is of particularimportance in maintaining authorship and dramatic pacing in theatricalor entertainment venues, and also for implementing controls over howmuch freedom a user 22 will have to navigate through the array 10.

[0086] In the present embodiment, the system 100 can be programmed suchthat certain portions of the array 10 are unavailable to the user 22 atspecified times or intervals. Thus, continuing with step 140 d of FIG.6, the server 18 makes the determination whether the user inputcorresponds to movement to the right through the array but is subject toa navigation control algorithm. The navigation control algorithm causesthe server 18 to determine, based upon navigation control factors,whether the user's desired movement is permissible.

[0087] More specifically, the navigation control algorithm, which isprogrammed in the server 18, determines whether the desired movementwould cause the current node address to fall outside the permissiblerange of node coordinates. In the present embodiment, the permissiblerange of node coordinates is predetermined and depends upon the time ofday, as noted by the server 18. Thus, in the present embodiment, thenavigation control factors include time. As will be appreciated by thoseskilled in the art, permissible camera nodes and control factors can becorrelated in a table stored in memory.

[0088] In an alternate embodiment, the navigation control factorsinclude time as measured from the beginning of a performance beingviewed, also as noted by the server. In such an embodiment, the systemoperator can dictate from where in the array a user will view certainscenes. In another alternate embodiment, the navigation control factoris speed of movement through the array. For example, the faster a user22 moves or navigates through the array, the wider the turns must be. Inother alternate embodiments, the permissible range of node coordinatesis not predetermined. In one embodiment, the navigation control factorsand, therefore, the permissible range, is dynamically controlled by thesystem operator who communicates with the server via an input device.

[0089] Having determined that the user input is subject to thenavigation control algorithm, the server 18 further proceeds, in step150 d, to increment the current node address along a predetermined path.By incrementing the current node address along a predetermined path, thesystem operator is able to corral or focus the attention of the user 22to the particular view of the permissible cameras 14, therebymaintaining authorship and dramatic pacing in theatrical andentertainment venues.

[0090] In an alternate embodiment where the user input is subject to anavigation control algorithm, the server 18 does not move the user alonga predetermined path. Instead, the server 18 merely awaits a permissibleuser input and holds the view at the current node. Only when the server18 receives a user input resulting in a permissible node coordinate willthe server 18 adjust the user's view.

[0091] Additional Source Output

[0092] In addition to moving through the array 10, the user 22 may, atpredetermined locations in the array 10, choose to leave the real worldenvironment being viewed. More specifically, additional source outputs,such as computer graphic imagery, virtual world imagery, applets, filmclips, and other artificial and real camera outputs, are made availableto the user 22. In one embodiment, the additional source output iscomposited with the view of the real environment. In an alternateembodiment, the user's view transfers completely from the realenvironment to that offered by the additional source output.

[0093] More specifically, the additional source output is stored(preferably in digital form) in the electronic storage device 20. Uponthe user 22 inputting a desire to view the additional source output, theserver 18 transmits the additional source output to the userinterface/display device 24. The present embodiment, the server 18simply transmits the additional source output to the user display device24. In an alternate embodiment, the server 18 first composites theadditional source output with the camera output and then transmits thecomposited signal to the user interface/display device 24.

[0094] As shown in step 140 e, the server 18 makes the determinationwhether the user input corresponds to moving in the array into thesource output. If the user 22 decides to move into the additional sourceoutput, the server 18 adjusts the view by substituting the additionalsource output for the updated camera output identified in either ofsteps 150 a-d.

[0095] Once the current node address is updated in either of steps 150a-d, the server 18 proceeds to adjust the user's view in step 250. Whenadjusting the view, the server 18 “mixes” the existing or current cameraoutput being displayed with the output of the camera 14 identified bythe updated camera node address. Mixing the outputs is achieveddifferently in alternate embodiments of the invention. In the presentembodiment, mixing the outputs involves electronically switching at aparticular speed from the existing camera output to the output of thecamera 14 having the new current node address.

[0096] It is to be understood that in this and other preferredembodiments disclosed herein, the camera outputs are synchronized. As iswell known in the art, a synchronizing signal from a “sync generator” issupplied to the cameras. The sync generator may take the form of thoseused in video editing and may comprise, in alternate embodiments, partof the server, the hub, and/or a separate component coupled to thearray.

[0097] As described above, at critical speed, the server 18 switchescamera outputs approximately at a rate of 24 per second, or one every0.04167 seconds. If the user 22 is moving through the array 10 at underthe critical speed, the outputs of the intermediate cameras 14 are eachdisplayed for a relatively longer duration than if the user is moving atthe critical speed. Similarly, each output is displayed for a relativelyshorter duration when a user navigates at over the critical speed. Inother words, the server 18 adjusts the switching speed based on thespeed of the movement through the array 10.

[0098] Of course, it is to be understood that in a simplified embodimentof the present invention, the user may navigate at only the criticalspeed.

[0099] In another alternate embodiment, mixing the outputs is achievedby compositing the existing or current output and the updated cameranode output. In yet another embodiment, mixing involves dissolving theexisting view into the new view. In still another alternate embodiment,mixing the outputs includes adjusting the frame refresh rate of the userdisplay device. Additionally, based on speed of movement through thearray, the server may add motion blur to convey the realistic sense ofspeed.

[0100] In yet another alternate embodiment, the server causes a blackscreen to be viewed instantaneously between camera views. Such anembodiment is analogous to blank film between frames in a movie reel.Furthermore, although not always advantageous, such black screens reducethe physiologic “carrying over” of one view into a subsequent view.

[0101] It is to be understood that the user inputs corresponding tomovements through the array at different speeds may include eitherdifferent keystrokes on a keypad, different positions of a joystick,positioning a joystick in a given position for a predetermined length oftime, and the like. Similarly, the decision to move into an additionalsource output may be indicated by a particular keystroke, joystickmovement, or the like.

[0102] In another embodiment, mixing may be accomplished by “mosaicing”the outputs of the intermediate cameras 14. U.S. Pat. No. 5,649,032entitled System For Automatically Aligning Images To Form A Mosaic Imageto Peter J. Burt et al. discloses a system and method for generating amosaic from a plurality of images and is hereby incorporated byreference. The server 18 automatically aligns one camera output toanother camera output, a camera output to another mosaic (generated-frompreviously occurring camera output) such that the output can be added tothe mosaic, or an existing mosaic to a camera output.

[0103] Once the mosaic alignment is complete, the present embodimentutilizes a mosaic composition process to construct (or update) a mosaic.The mosaic composition comprises a selection process and a combinationprocess. The selection process automatically selects outputs forincorporation into the mosaic and may include masking and croppingfunctions to select the region of interest in a mosaic. Once theselection process selects which output(s) are to be included in themosaic, the combination process combines the various outputs to form themosaic. The combination process applies various output processingtechniques, such as merging, fusing, filtering, output enhancement, andthe like, to achieve a seamless combination of the outputs. Theresulting mosaic is a smooth view that combines the constituent outputssuch that temporal and spatial information redundancy are minimized inthe mosaic. In one embodiment of the present invention, the mosaic maybe formed as the user moves through the system (on the fly) and theoutput image displayed close to real time. In another embodiment, thesystem may form the mosaic from a predetermined number of outputs orduring a predetermined time interval, and then display the imagespursuant to the user's navigation through the environment.

[0104] In yet another embodiment, the server 18 enables the output to bemixed by a “tweening” process. One example of the tweening process isdisclosed in U.S. Pat. No. 5,529,040 entitled Method For DeterminingSensor Motion And Scene Structure And Image Processing System Thereforto Keith J. Hanna, herein incorporated by reference. Tweening enablesthe server 18 to process the structure of a view from two or more cameraoutputs of the view.

[0105] Applying the Hanna patent to the telepresence method/systemherein, tweening is now described. The server monitors the movementamong the intermediate cameras 14 through a scene using local scenecharacteristics such as brightness derivatives of a pair of cameraoutputs. A global camera output movement constraint is combined with alocal scene characteristic constancy constraint to relate local surfacestructures with the global camera output movement model and local scenecharacteristics. The method for determining a model for global cameraoutput movement through a scene and scene structure model of the scenefrom two or more outputs of the scene at a given image resolutioncomprises the following steps:

[0106] (a) setting initial estimates of local scene models and a globalcamera output movement model;

[0107] (b) determining a new value of one of the models by minimizingthe difference between the measured error in the outputs and the errorpredicted by the model;

[0108] (c) resetting the initial estimates of the local scene models andthe image sensor motion model using the new value of one of the modelsdetermined in step (b);

[0109] (d) determining a new value of the second of the models using theestimates of the models determined in step (b) by minimizing thedifference between the measured error in the outputs and the errorpredicted by the model;

[0110] (e) warping one of the outputs towards the other output using thecurrent estimates of the models at the given image resolution; and

[0111] (f) repeating steps (b), (c), (d) and (e) until the differencesbetween the new values of the models and the values determined in theprevious iteration are less than a certain value or until a fixed numberof iterations have occurred.

[0112] It should be noted that where the Hanna patent effectuates thetweening process by detecting the motion of an image sensor (e.g., avideo camera), an embodiment of the present invention monitors the usermovement among live cameras or storage nodes.

[0113] In an alternate embodiment, although not always necessary, toensure a seamless progression of views, the server 18 also transmits tothe user display device 24 outputs from some or all of the intermediatecameras, namely those located between the current camera node and theupdated camera node. Such an embodiment will now be described withreference to FIGS. 7a-7 g. Specifically, FIG. 7a illustrates acurvilinear portion of an array 10 that extends along the X axis or tothe left and right from the user's perspective. Thus, the coordinatesthat the server 18 associates with the cameras 14 differ only in the Xcoordinate. More specifically, for purposes of the present example, thecameras 14 can be considered sequentially numbered, starting with theleft-most camera 14 being the first, i.e., number “1”. The X coordinateof each camera 14 is equal to the camera's position in the array. Forillustrative purposes, particular cameras will be designate 14-X, whereX equals the camera's position in the array 10 and, thus, its associatedX coordinate.

[0114] In general, FIGS. 7a-7 g illustrate possible user movementthrough the array 10. The environment to be viewed includes threeobjects 602, 604, 606, the first and second of which include numberedsurfaces. As will be apparent, these numbered surface allow a betterappreciation of the change in user perspective.

[0115] In FIG. 7a, six cameras 14-2, 14-7, 14-11, 14-14, 14-20, 14-23 ofthe array 10 are specifically identified. The boundaries of eachcamera's view is identified by the pair of lines 14-2 a, 14-7 a, 14-11a, 14-14 a, 14-20 a, 14-23 a, radiating from each identified camera14-2, 14-7, 14-11, 14-14, 14-20, 14-23, respectively. As describedbelow, in the present example the user 22 navigates through the array 10along the X axis such that the images or views of the environment arethose corresponding to the identified cameras 14-2, 14-7, 14-11, 14-14,14-20, 14-23.

[0116] The present example provides the user 22 with the starting viewfrom camera 14-2. This view is illustrated in FIG. 7b. The user 22,desiring to have a better view of the object 702, pushes the “7” key onthe keyboard. This user input is transmitted to and interpreted by theserver 18.

[0117] Because the server 18 has been programmed to recognized the “7”key as corresponding to moving or jumping through the array to camera14-7. The server 18 changes the X coordinate of the current camera nodeaddress to 7, selects the output of camera 14-7, and adjusts the view orimage sent to the user 22. Adjusting the view, as discussed above,involves mixing the outputs of the current and updated camera nodes.Mixing the outputs, in turn, involves switching intermediate cameraoutputs into the view to achieve the seamless progression of thediscrete views of cameras 14-2 through 14-7, which gives the user 22 thelook and feel of moving around the viewed object. The user 22 now hasanother view of the first object 702. The view from camera 14-7 is shownin FIG. 7c. As noted above, if the jump in camera nodes is greater thana predetermined limit, the server 18 would omit some or all of theintermediate outputs.

[0118] Pressing the “right arrow” key on the keyboard, the user 22indicates to the system 100 a desire to navigate to the right atcritical speed. The server 18 receives and interprets this user input asindicating such and increments the current camera node address by n=4.Consequently, the updated camera node address is 14-11. The server 18causes the mixing of the output of camera 14-11 with that of camera14-7. Again, this includes switching into the view the outputs of theintermediate cameras (i.e., 14-8, 14-9, and 14-10) to give the user 22the look and feel of navigating around the viewed object. The user 22 isthus presented with the view from camera 14-11, as shown in FIG. 7d.

[0119] Still interested in the first object 702, the user 22 enters auser input, for example, “alt-right arrow,” indicating a desire to moveto the right at less than critical speed. Accordingly, the server 18increments the updated camera node address by n−1 nodes, namely 3 in thepresent example, to camera 14-14. The outputs from cameras 14-11 and14-14 are mixed, and the user 22 is presented with a seamless viewassociated with cameras 14-11 through 14-14. FIG. 7e illustrates theresulting view of camera 14-14.

[0120] With little to see immediately after the first object 702, theuser 22 enters a user input such as “shift-right arrow,” indicating adesire to move quickly through the array 10, i.e., at over the criticalspeed. The server 18 interprets the user input and increments thecurrent node address by n+2, or 6 in the present example. The updatednode address thus corresponds to camera 14-20. The server 18 mixes theoutputs of cameras 14-14 and 14-20, which includes switching into theview the outputs of the intermediate cameras 14-15 through 14-19. Theresulting view of camera 14-20 is displayed to the user 22. As shown inFIG. 7f, the user 22 now views the second object 704.

[0121] Becoming interested in the third object 704, the user 22 desiresto move slowly through the array 10. Accordingly, the user 22 enters“alt-right arrow” to indicate moving to the right at below criticalspeed. Once the server 18 interprets the received user input, it updatesthe current camera node address along the X axis by 3 to camera 14-23.The server 18 then mixes the outputs of camera 14-20 and 14-23, therebyproviding the user 22 with a seamless progression of views throughcamera 14-23. The resulting view 14-23 a is illustrated in FIG. 7g.

[0122] Other Data Devices

[0123] It is to be understood that devices other than cameras may beinterspersed in the array. These other devices, such as motion sensorsand microphones, provide data to the server(s) for processing. Forexample, in alternate embodiments output from motion sensors ormicrophones are fed to the server(s) and used to scale the array. Morespecifically, permissible camera nodes (as defined in a table stored inmemory) are those near the sensor or microphone having a desired outpute.g., where there is motion or sound. As such, navigation controlfactors include output from other such devices. Alternatively, theoutput from the sensors or microphones are provided to the user.

[0124] An alternate embodiment in which the array of cameras includesmultiple microphones interspersed among the viewed environment and thecameras will now be described with reference to FIG. 8. The system 800generally includes an array of cameras 802 coupled to a server 804,which, in turn, is coupled to one or more user interface and displaydevices 806 and an electronic storage device 808. A hub 810 collects andtransfers the outputs from the array 802 to the server 804. Morespecifically, the array 802 comprises modular rails 812 that areinterconnected. Each rail 812 carries multiple microcameras 814 and amicrophone 816 centrally located at rail 812. Additionally, the system800 includes microphones 818 that are physically separate from the array802. The outputs of both the cameras 814 and microphones 816, 818 arecoupled to the server 804 for processing.

[0125] In general, operation of the system 800 proceeds as describedwith respect to system 100 of FIGS. 1-2 d and 5-6. Beyond the operationof the previously described system 100, however, the server 804 receivesthe sound output from the microphones 816, 818 and, as with the cameraoutput, selectively transmits sound output to the user. As the server804 updates the current camera node address and changes the user's view,it also changes the sound output transmitted to the user. In the presentembodiment, the server 804 has stored in memory an associated range ofcamera nodes with a given microphone, namely the cameras 814 on eachrail 810 are associated with the microphone 816 on that particular rail810. In the event a user attempts to navigate beyond the end of thearray 802, the server 804 determines the camera navigation isimpermissible and instead updates the microphone node output to that ofthe microphone 818 adjacent to the array 802.

[0126] In an alternate embodiment, the server 804 might include adatabase in which camera nodes in a particular area are associated witha given microphones. For example, a rectangle defined by the (X, Y, Z)coordinates (0,0,0), (10,0,0), (10,5,0), (0,5,0), (0,0,5), (10,0,5),(10,5,5) and (0,5,5) are associated with a given microphone. It is to beunderstood that selecting one of the series of microphones based on theuser's position (or view) in the array provides the user with a soundperspective of the environment that coincides with the visualperspective.

[0127] It is to be understood that the server of the embodimentsdiscussed above may take any of a number of known configurations. Twoexamples of server configurations suitable for use with the presentinvention will be described with reference to FIGS. 9 and 10. Turningfirst to FIG. 9, the server 902, electronic storage device 20, array 10,users (1,2,3, . . . N) 22-1-22-N, and associated user interface/displaydevices 24-1-24-N are shown therein.

[0128] The server 902 includes, among other components, a processingmeans in the form of one or more central processing units (CPU) 904coupled to associated read only memory (ROM) 906 and a random accessmemory (RAM) 908. In general, ROM 906 is for storing the program thatdictates the operation of the server 902, and the RAM 908 is for storingvariables and values used by the CPU 904 during operation. Also coupledto the CPU 904 are the user interface/display devices 24. It is to beunderstood that the CPU may, in alternate embodiments, comprise severalprocessing units, each performing a discrete function.

[0129] Coupled to both the CPU 904 and the electronic storage device 20is a memory controller 910. The memory controller 910, under directionof the CPU 904, controls accesses (reads and writes) to the storagedevice 20. Although the memory controller 910 is shown as part of theserver 902, it is to be understood that it may reside in the storagedevice 20.

[0130] During operation, the CPU 904 receives camera outputs from thearray 10 via bus 912. As described above, the CPU 904 mixes the cameraoutputs for display on the user interface/display device 24. Whichoutputs are mixed depends on the view selected by each user 22.Specifically, each user interface/display devices 24 transmits acrossbus 914 the user inputs that define the view to be displayed. Once theCPU 904 mixes the appropriate outputs, it transmits the resulting outputto the user interface/display device 24 via bus 916. As shown, in thepresent embodiment, each user 22 is independently coupled to the server902.

[0131] The bus 912 also carries the camera outputs to the storage device20 for storage. When storing the camera outputs, the CPU 904 directs thememory controller 910 to store the output of each camera 14 in aparticular location of memory in the storage device 20.

[0132] When the image to be displayed has previously been stored in thestorage device 20, the CPU 904 causes the memory controller 910 toaccess the storage device 20 to retrieve the appropriate camera output.The output is thus transmitted to the CPU 904 via bus 918 where it ismixed. Bus 918 also carries additional source output to the CPU 904 fortransmission to the users 22. As with outputs received directly from thearray 10, the CPU 904 mixes these outputs and transmits the appropriateview to the user interface/display device 24.

[0133]FIG. 10 shows a server configuration according to an alternateembodiment of the present invention. As shown therein, the server 1002generally comprises a control central processing unit (CPU) 1004, amixing CPU 1006 associated with each user 22, and a memory controller1008. The control CPU 1004 has associated ROM 1010 and RAM 1012.Similarly, each mixing CPU 1006 has associated ROM 1014 and RAM 1016.

[0134] To achieve the functionality described above, the camera outputsfrom the array 10 are coupled to each of the mixing CPUs 1 through N1006-1, 1006-N via bus 1018. During operation, each user 22 entersinputs in the interface/display device 24 for transmission (via bus1020) to the control CPU 1004. The control CPU 1004 interprets theinputs and, via buses 1022-1, 1022-N, transmits control signals to themixing CPUs 1006-1, 1006-N instructing them which camera outputsreceived on bus 1018 to mix. As the name implies, the mixing CPUs1006-1, 1006-N mix the outputs in order to generate the appropriate viewand transmit the resulting view via buses 1024-1, 1024-N to the userinterface/display devices 24-1, 24-N.

[0135] In an alternate related embodiment, each mixing CPU 1006multiplexes outputs to more than one user 22. Indications of whichoutputs are to mixed and transmitted to each user 22 comes from thecontrol CPU 1004.

[0136] The bus 1018 couples the camera outputs not only to the mixingCPUs 1006-1, 1006-N, but also to the storage device 20. Under control ofthe memory controller 1008, which in turn is controlled by the controlCPU 1004, the storage device 20 stores the camera outputs in knownstorage locations. Where user inputs to the control CPU 1004 indicate ausers' 22 desire to view stored images, the control CPU 1004 causes thememory controller 1008 to retrieve the appropriate images from thestorage device 20. Such images are retrieved into the mixing CPUs 1006via bus 1026. Additional source output is also retrieved to the mixingCPUs 1006-1, 1006-N via bus 1026. The control CPU 1004 also passescontrol signals to the mixing CPUs 1006-1, 1006-N to indicate whichoutputs are to be mixed and displayed.

[0137] Stereoscopic Views

[0138] It is to be understood that it is within the scope of the presentinvention to employ stereoscopic views of the environment. To achievethe stereoscopic view, the system retrieves from the array (or theelectronic storage device) and simultaneously transmits to the user atleast portions of outputs from two cameras. The server processingelement mixes these camera outputs to achieve a stereoscopic output.Each view provided to the user is based on such a stereoscopic output.In one stereoscopic embodiment, the outputs from two adjacent cameras inthe array are used to produce one stereoscopic view. Using the notationof FIGS. 7a-7 g, one view is the stereoscopic view from cameras 14-1 and14-2. The next view is based on the stereoscopic output of cameras 14-2and 14-3 or two other cameras. Thus, in such an embodiment, the user isprovided the added feature of a stereoscopic seamless view of theenvironment.

[0139] Multiple Users

[0140] As described above, the present invention allows multiple usersto simultaneously navigate through the array independently of eachother. To accommodate multiple users, the systems described abovedistinguish between inputs from the multiple users and selects aseparate camera output appropriate to each user's inputs. In one suchembodiment, the server tracks the current camera node address associatedwith each user by storing each node address in a particular memorylocation associate with that user. Similarly, each user's input isdifferentiated and identified as being associated with the particularmemory location with the use of message tags appended to the user inputsby the corresponding user interface device.

[0141] In an alternate embodiment, two or more users may choose to belinked, thereby moving in tandem and having the same view of theenvironment. In such an embodiment, each includes identifying anotheruser by his/her code to serve as a “guide”. In operation, the serverprovides the outputs and views selected by the guide user to both theguide and the other user selecting the guide. Another user input causesthe server to unlink the users, thereby allowing each user to controlhis/her own movement through the array.

[0142] Multiple Arrays

[0143] In certain applications, a user may also wish to navigate forwardand backward through the environment, thereby moving closer to orfurther away from an object. Although it is within the scope of thepresent invention to use cameras with zoom capability, simply zoomingtowards an object does not change the user's image point perspective.One such embodiment in which users can move dimensionally forward andbackward through the environment with a changing image point perspectivewill now be described with respect to FIG. 11 and continuing referenceto FIG. 1. As will be understood by those skilled in the art, the arraysdescribed with reference to FIG. 11 may be used with any server, storagedevice and user terminals described herein.

[0144]FIG. 11 illustrates a top plan view of another embodiment enablingthe user to move left, right, up, down, forward or backwards through theenvironment. A plurality of cylindrical arrays (121-1-121-n) ofdiffering diameters comprising a series of cameras 14 may be situatedaround an environment comprising one or more objects 1200, onecylindrical array at a time. Cameras 14 situated around the object(s)1100 are positioned along an X and Z coordinate system. Accordingly, anarray 12 may comprise a plurality of rings of the same circumferencepositioned at different positions (heights) throughout the z-axis toform a cylinder of cameras 14 around the object(s) 1100. This alsoallows each camera in each array 12 to have an associated, uniquestorage node address comprising an X and Z coordinate—i.e., array1(X,Z). In the present embodiment, for example, a coordinate valuecorresponding to an axis of a particular camera represents the number ofcamera positions along that axis the particular camera is displaced froma reference camera. In the present embodiment, from the user'sperspective, the X axis runs around the perimeter of an array 12, andthe Z axis runs down and up. Each storage node is associated with acamera view identified by its X, Z coordinate.

[0145] As described above, the outputs of the cameras 14 are coupled toone or more servers for gathering and transmitting the outputs to theserver 18.

[0146] In one embodiment, because the environment is static, each camerarequires only one storage location. The camera output may be stored in alogical arrangement, such as a matrix of n arrays, wherein each arrayhas a plurality of (X,Z) coordinates. In one embodiment, the nodeaddresses may comprise of a specific coordinate within an array—i.e.,Array₁(X_(n), Z_(n)), Array₂(X_(n), Z_(n)) through Array_(n)(X_(n),Z_(n)) . As described below, users can navigate the stored images inmuch the same manner as the user may navigate through an environmentusing live camera images.

[0147] The general operation of one embodiment of inputting images instorage device 20 for transmission to a user will now be described withreference to FIG. 12 and continuing reference to FIG. 11. As shown instep 1210, a cylindrical array 12-1 is situated around the object(s)located in an environment 1100. The view of each camera 14 istransmitted to server 18 in step 1220. Next, in step 1220, theelectronic storage device 20 of the server 18 stores the output of eachcamera 14 at the storage node address associated with that camera 14.Storage of the images may be effectuated serially, from one camera 14 ata time within the array 12, or by simultaneous transmission of the imagedata from all of the cameras 14 of each array 12. Once the output foreach camera 14 of array 12-1 is stored, cylindrical array 12-1 isremoved from the environment (step 1240). In step 1250, a determinationis made as to the availability of additional cylindrical arrays 12 ofdiffering diameters to those already situated. If additional cylindricalarrays 12 are desired, the process repeats beginning with step 1210.When no additional arrays 12 are available for situating around theenvironment, the process of inputting images into storage devices 20 iscomplete (step 1260). At the end of the process, a matrix of addressablestored images exist.

[0148] Upon storing all of the outputs associated with the arrays 12-1through 12-n, a user may navigate through the environment. Navigation iseffectuated by accessing the input of the storage nodes by a userinterface device 24. In the present embodiment, the user inputsgenerally include moving around the environment or object 1100 by movingto the left or right, moving higher or lower along the z-axis, movingthrough the environment closer or further from the object 1100, or somecombination of moving around and through the environment. For example, auser may access the image stored in the node address Array₃(0,0) to viewan object from the camera previously located at coordinate (0,0) ofArray₃. The user may move directly forward, and therefore closer to theobject 1100, by accessing the image stored in Array₂(0,0) and thenArray₁(0,0). To move further away from the object and to the right andup, the user may move from the image stored in node address Array₁(0,0)and access the images stored in node address Array₂(1,1), followed byaccessing the image stored in node address Array₃(2,2), an so on. A usermay, of course, move among arrays and/or coordinates by any incrementschanging the point perspective of the environment with each node.Additionally, a user may jump to a particular camera view of theenvironment. Thus, a user may move throughout the environment in amanner similar to that described above with respect to accessing outputof live cameras. This embodiment, however, allows user to access imagesthat are stored in storage nodes as opposed to accessing live cameras.Moreover, this embodiment provides a convenient system and method toallow a user to move forward and backward in an environment.

[0149] It should be noted that although each storage node is associatedwith a camera view identified by its X, Z coordinate of a particulararray, other methods of identifying camera views and storage nodes canbe used. For example, other coordinate systems, such as those notingangular displacement from a fixed reference point as well as coordinatesystems that indicate relative displacement from the current camera nodemay be used. It should also be understood that the camera arrays 12 maybe other shapes other than cylindrical. Moreover, it is not essential,although often advantageous, that the camera arrays 12 surround theentire environment.

[0150] It is to be understood that the foregoing user inputs, namely,move clockwise, move counter-clockwise, up, down, closer to theenvironment, and further from the environment, are merely generaldescriptions of movement through the environment. Although the presentinvention is not so limited, in the present preferred embodiment,movement in each of these general directions is further defined basedupon the user input. Moreover the output generated by the server to theuser may be mixed when moving among adjacent storage nodes associatedwith environment views (along the x axis, z axis, or among juxtaposedarrays) to generate seamless movement throughout the environment. Mixingmay be accomplished by, but are not limited to, the processes describedabove.

[0151] Embodiments Covered

[0152] Although the present invention has been described in terms ofcertain preferred embodiments, other embodiments that are apparent tothose of ordinary skill in the art are also intended to be within thescope of this invention. Accordingly, the scope of the present inventionis intended to be limited only by the claims appended hereto.

1. A telepresence system for providing a first user with a first displayof an environment and a second user with a second display of theenvironment, the system comprising: an array of cameras, each camerahaving an associated view of the environment and an associated cameraoutput representing the associated view, the array including at leastone camera path and a first camera having a first output and a secondcamera having a second output; a first user interface device associatedwith the first user having first user inputs associated with movementalong a first path in the array; a second user interface deviceassociated with the second user having second user inputs associatedwith movement along a second path in the array; at least one processingelement coupled to the user interface devices for receiving user inputs,the processing element configured to interpret received first inputs andselect outputs of cameras in the first path, interpret received secondinputs and select outputs of cameras in the second path independently ofthe first inputs, thereby allowing the first user and second user tonavigate simultaneously and independently through the array, and theprocessing element is configured to mix the first and second outputs inaccordance with the received first user inputs by mosaicing the viewassociated with the first camera with the view of the second camera. 2.The system of claim 1 wherein the system further includes a memorystoring additional source output, wherein the user inputs include anindication of viewing the additional source output, and the processingelement is further configured to mix camera output and the additionalsource output upon receiving the indication to view the additionalsource output by mosaicing the camera output and the additional sourceoutput.
 3. The system of claim 2 wherein the additional source outputincludes output from the group of outputs including: computer graphicimagery, virtual world imagery, applets, film clips, and animation.
 4. Adevice for providing a user with a display of an environment in responseto user inputs, the system comprising: an array of cameras, each camerahaving an associated view of the environment and an associated cameraoutput representing the associated view; memory storing an additionalsource output; and at least one processing element coupled to the memoryfor receiving the additional source output, the processing elementconfigured to interptet user inputs and select, based on the userinputs, a camera output to provide to the user, the additional sourceoutput to provide to the user, or both a camera view and the additionalsource output to provide to the user, and the processing element isconfigured to mix the additional source output with camera output bymosaicing from the camera output to the additional source output.
 5. Asystem for remote seamless viewing of an environment from an array ofcameras, each having an output representing an image of the environment,the device comprising: an interface device having inputs for selecting apath through at least a portion of the array from which to view theenvironment, the path including a sequence of cameras, each camera inthe sequence having a different point perspective and a field of viewthat overlaps that of an adjacent cameras; and a display device forsequentially displaying the image from each camera in the sequence bymosaicing the image of a current camera in the sequence to the image ofa next camera in the sequence, thereby providing the user a seamlessview of the environment.
 6. A method for seamless viewing of anenvironment, the method comprising: receiving electronically a firstimage from an array of cameras, the first image having a first field ofview; receiving electronically a second image from the array, the secondimage having a second field of view that overlaps the first field ofview; receiving electronically a third image from the array, the thirdimage having a third field of view that overlaps the second field ofview; mosaicing the first image with the second image and then mosaicingthe second image with the third image; and displaying the first, second,third and mosaic images in sequence to obtaining a seamless view throughthe environment.
 7. The method of claim 6 further including selecting anadditional source output to be displayed and mosaicing the additionalsource output with the third image.
 8. The method of claim 6 wherein thefirst, second and third images correspond to first, second, and thirdcameras, respectively.
 9. The method of claim 8 wherein the secondcamera is adjacent to the first and third cameras.
 10. A telepresencesystem for providing a first user with a first display of an environmentand a second user with a second display of the environment, the systemcomprising: an array of cameras, each camera having an associated viewof the environment and an associated camera output representing theassociated view, the array including at least one camera path and afirst camera having a first output and a second camera having a secondoutput; a first user interface device associated with the first userhaving first user inputs associated with movement along a first path inthe array; a second user interface device associated with the seconduser having second user inputs associated with movement along a secondpath in the array; at least one processing element coupled to the userinterface devices for receiving user inputs, the processing elementconfigured to interpret received first inputs and select outputs ofcameras in the first path, interpret received second inputs and selectoutputs of cameras in the second path independently of the first inputs,thereby allowing the first user and second user to navigatesimultaneously and independently through the array, and the processingelement is configured to mix the first and second outputs in accordancewith the received first user inputs by tweening the view associated withthe first camera with the view of the second camera.
 11. The system ofclaim 10 wherein the system further includes a memory storing additionalsource output, wherein the user inputs include an indication of viewingthe additional source output, and the processing element is furtherconfigured to mix camera output and the additional source output uponreceiving the indication to view the additional source output bytweening the camera output and the additional source output.
 12. Thesystem of claim 11 wherein the additional source output includes outputfrom the group of outputs including: computer graphic imagery, virtualworld imagery, applets, film clips, and animation.
 13. A device forproviding a user with a display of an environment in response to userinputs, the system comprising: an array of cameras, each camera havingan associated view of the environment and an associated camera outputrepresenting the associated view; memory storing an additional sourceoutput; and at least one processing element coupled to the memory forreceiving the additional source output, the processing elementconfigured to interpret user inputs and select, based on the userinputs, a camera output to provide to the user, the additional sourceoutput to provide to the user, or both a camera view and the additionalsource output to provide to the user, and the processing element isconfigured to mix the additional source output with camera output bytweening from the camera output to the additional source output.
 14. Asystem for remote seamless viewing of an environment from an array ofcameras, each having an output representing an image of the environment,the device comprising: an interface device having inputs for selecting apath through at least a portion of the array from which to view theenvironment, the path including a sequence of cameras, each camera inthe sequence having a different point perspective and a field of viewthat overlaps that of an adjacent cameras; and a display device forsequentially displaying the image from each camera in the sequence bytweening the image of a current camera in the sequence to the image of anext camera in the sequence, thereby providing the user a seamless viewof the environment.
 15. A method for seamless viewing of an environment,the method comprising: receiving electronically a first image from anarray of cameras, the first image having a first field of view;receiving electronically a second image from the array, the second imagehaving a second field of view that overlaps the first field of view;receiving electronically a third image from the array, the third imagehaving a third field of view that overlaps the second field of view;tweening the first image with the second image and then mosaicing thesecond image with the third image; and displaying the first, second,third and tweened images in sequence to obtaining a seamless viewthrough the environment.
 16. The method of claim 15 further includingselecting an additional source output to be displayed and tweening theadditional source output with the third image.
 17. The method of claim15 wherein the first, second and third images correspond to first,second, and third cameras, respectively.
 18. The method of claim 17wherein the second camera is adjacent to the first and third cameras.19. A method of providing users with real time views of a remoteenvironment, the method comprising: receiving electronic images of theenvironment from an array of cameras, the array including at least onecamera path through the environment; receiving a first input -from afirst-user interface device associated with a first user, the firstinput indicating movement along a first path; receiving a second inputfrom a second user interface device associated with a second user, thesecond input indicating movement along a second path; obtaining a firstmixed image by mosaicing or tweening, with a first processing element, afirst image with a second image in accordance with the first input;obtaining a second mixed image by mosaicing or tweening, with a secondprocessing element, a third image with a fourth image in accordance withthe second input; providing the first user with the first mixed image insubstantially real time, thereby simulating movement along the firstpath; and providing the second user with the second mixed imagesubstantially in real time and simultaneous to providing the first userwith the first mixed image, thereby independently and simulatingmovement along the second path.
 20. The method of claim 19 whereinreceiving the first input includes receiving an indication of mosaicingor tweening an additional source output, the method further includingobtaining a third mixed output by mosaicing or tweening the second imagewith the additional source output.
 21. The method of claim 19 furtherincluding obtaining a third mixed image by mosaicing or tweening thesecond image with a fifth image in accordance with the first input andproviding the first use the third mixed image.
 22. A telepresence systemfor providing a first user with a first display of an environment and asecond user with a second display of the environment, the systemcomprising: a plurality of removable arrays of cameras, each camerahaving an associated view of the environment and an associated cameraoutput representing the associated view; at least one storage deviceincluding a plurality of storage nodes wherein the output of each camerais stored in an associated storage node, the storage nodes areaccessible to permit at least one path for viewing the environment; afirst user interface device associated with the first user having firstuser inputs associated with movement along a first path in theenvironment; a second user interface device associated with the seconduser having second user inputs associated with movement along a secondpath in the environment; at least one processing element coupled to theuser interface devices for receiving user inputs including moving updown, clockwise around an environment, counter-clockwise around anenvironment, forward and backward through the environment, theprocessing element configured to interpret received first inputs andselect outputs of the storage nodes forming the first path, andinterpret received second inputs and select outputs of storage nodesforming the second path independently of the first inputs, therebyallowing the first user and second user to navigate simultaneously andindependently through the environment.
 23. The telepresence system ofclaim 22 wherein the outputs of the cameras are accessible by theprocessing element.
 24. The telepresence system of claim 22 wherein eachremovable array is situated at different lengths from the environment.25. The telepresence system of claim 23 wherein each array is removedafter the cameras in the array have transmitted the output to theassociated storage node.
 26. The telepresence system of claim 25 whereineach array is of cylindrical shape and of a varying diameter.
 27. Atelepresence system for providing a first user with a first display ofan environment and a second user with a second display of theenvironment, the system comprising: a plurality of removable arrays ofcameras, each camera having an associated view of the environment and anassociated camera output representing the associated view, the arrayssituated at varying lengths from the environment and including at leastone path for viewing the environment, said arrays are removed after thecameras in the array have transmitted the output to an associatedstorage node; at least one storage device including a plurality ofstorage nodes wherein the output of each camera is stored in anassociated storage node, the storage nodes are accessible to permit atleast one path for viewing the environment; a first user interfacedevice associated with the first user having first user inputsassociated with movement along a first path in the environment; a seconduser interface device associated with the second user having second userinputs associated with movement along a second path in the environment;at least one processing element coupled to the user interface devicesfor receiving user inputs including moving up down, clockwise around anenvironment, counter-clockwise around an environment, forward andbackward through the environment, the processing element configured tointerpret received first inputs and select outputs of the storage nodeforming the first path, and interpret received second inputs and selectoutputs of the storage node forming the second path independently of thefirst inputs, thereby allowing the first user and second user tonavigate simultaneously and independently through the environment.
 28. Amethod of providing users with views of a remote environment, the methodcomprising: receiving electronic images of the environment from aplurality of array of cameras; storing the image of the environment instorage nodes associated with each camera, the storage nodes areaccessible to permit at least one path for viewing the environment;removing the array of cameras after storing the image in the associatedstorage node; receiving a first input from a first user interface deviceassociated with a first user, the first input indicating movement alonga first path; receiving a second input from a second user interfacedevice associated with a second user, the second input indicatingmovement along a second path; obtaining a first mixed image by mixing,with a first processing element, a first image with a second image inaccordance with the first input; obtaining a second mixed image bymixing, with a second processing element, a third image with a fourthimage in accordance with the second input; providing the first user withthe first mixed image thereby simulating movement along the first path;and providing the second user with the second mixed image therebyindependently and simulating movement along the second path.
 29. Themethod of claim 28 wherein one array at a time is situated around theenvironment.
 30. The method of claim 29 wherein the storage nodes may beaccessed to permit a plurality of navigable paths and the first pathdiffers from the second path.